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ABSTRACT 

The purpose of this study was to define and validate a set of criteria that would 
conceptualize the “big picture” operation of CTE within the state. Named the operational 
infrastructure of secondary CTE (OISCTE), these statements were defined as the basic, 
underlying framework of criteria that loccd, state, and university leaders in CTE must develop, 
grow, and/or maintain in order to effectively and efficiently operate a statewide CTE program. 
The researchers examined the extant literature and formulated 13 criteria statements that 
embodied the OISCTE. Then, using a survey technique and veteran local and state CTE 
administrators and university teacher educators, the researchers validated the OISCTE. The 
OISCTE has policy as well as assessment applications of which the researchers explcdn within 
the manuscript. 

Introduction 

The profession of career and technical education (CTE) at the secondary level has 
become increasingly complicated. The need or requirement of CTE agencies (local, state, and 
university) to measure progress, synthesize responsibilities, and integrate disciplines has never 
been more prominent, nor emphasized. State and local CTE agencies are required in the form of 
new legislation, regulations, and procedures to take on this administrative burden with, in many 
instances, dwindling budgets. This has led to the uncomfortable, but all too common, practice of 
decreasing or eliminating support for basic operational criteria such as the ones defined and 
validated in this piece of research. For example, the CTE state-level administration in one of the 
researcher’s state has eliminated certain teacher education grants in order to help fund 
accountability measures put forth by Perkins IV. These teacher education grants were generally 
used to help improve and maintain CTE teacher education programs and/or recruit quality CTE 
teacher candidates. As Dr. Richard Lynch, CTE teacher educator, once stated, “...without good 
teachers, you cannot have good instruction nor good programs. The studies on the education and 
re-education of good teachers.. .indicate that effective teacher education takes time and it takes 
money”(NCRVE, 1991, p. 23). Yet, as state and local budgets decrease, some programs and/or 
initiatives must take the brunt of fiscal belt tightening. Of course, these programs and/or 
initiatives are generally not the ones that new federal and state guidelines or regulations are 
requiring state and/or local CTE agencies to increase or emphasize. What effect does this have 
on the general operation of secondary-level CTE? While no new legislation or accountability 
measure can account for all of the factors, what does such a change do to the objectives, tasks, 
programs, and/or initiatives not included in the new legislation or measure? 
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First and foremost, CTE must identify and agree upon the criteria that make up the 
general operation of secondary-level CTE. The researchers attempted to accomplish this task 
with the development and validation of the Operational Infrastructure of Secondary-level CTE 
(OISCTE). The OISCTE is defined as the basic, underlying framework of criteria that state 
leaders in CTE (state and local administrators and university-level teacher educators) must 
develop, grow, and/or maintain in order to effectively and efficiently operate a statewide CTE 
program. Therefore, the purpose of this study was to define and validate the set of criteria that 
would embody the OISCTE and thus answer the question: What is the OISCTE? 

Methodology 

In order to answer the research question, the researchers conducted a literature review to 
aid in the development of the OISCTE’s criteria statements. The literature review involved 
synthesizing national and state documents related to the regulation, administration, and 
evaluation of CTE programs at all levels. Additionally, it required contacting the current Virginia 
CTE state director for insight into their role within the operation of secondary-level CTE. 

Once the criteria statements were established, the researchers validated them using a 
survey that was completed by local and state CTE administrators as well as university-level CTE 
teacher educators from Virginia. The survey utilized a Likert scale ( r=5) to identify their level of 
agreement as to the inclusion of the criteria statements derived from the literature review. 
Additionally, the participants were given an opportunity to add comments concerning their 
ratings or the criteria statements. It is important to note that this survey was included in Round 1 
of a three-round Policy Delphi. The purpose of the Policy Delphi was to identify the impact of a 
federal formula change that occurred with the passage of the 1990 Perkins Act on the OISCTE 
within Virginia (Manley, 2011). Before the subsequent rounds could proceed, the participants 
needed to validate the set of criteria that embodied the OISCTE so they could begin the process 
of identifying how the federal funding change affected each OISCTE criterion statement. 

Upon participant completion of the survey, the researchers calculated the frequency, 
median, mean, standard deviation, and interquartile range (IQR) for each criteria statement. 
Additionally, the researchers analyzed participants’ comments for relevancy. For a criterion 
statement to be considered validated, the statement’s mean was required to be at least 4.0 on a 
five-point scale with 5.0 denoting strong agreement as to the inclusion of the criterion statement. 

Participant Selection 

Since the study that included this survey sought to identify the impact of a policy change 
that occurred in the early 1990s, the participants must have been working as either a local, state, 
or university-level CTE administrator or teacher educator within Virginia from 1989 to 1993. 

The researchers used multiple resources (state library documents, online phone and email 
databases, university course catalogs, AVA and ACTE rosters, and interviews) to identify and 
locate potential participants. 

Conceptual Framework 
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The OISCTE is grounded in criteria related to the development of competency-based 
education (CBE). More specifically, the development of OISCTE followed pertinent 
specification criteria for describing and assessing competency-based programs as noted by 
Burke, Hansen, Houston, and Johnson (as cited in Tuxworth, 1989). First, Burke et al. noted that 
competency statements must be . .based on an analysis of the professional role(s) and/or a 
theoretical formulation of professional responsibilities” (p.12). This criterion was the basis for 
reviewing the literature related to the statewide operation of secondary-level CTE. In addition, 
the Burke et al. note that competency statements must “...facilitate criterion referenced 
assessment” (p.12). This criterion helped researchers determine the construction the OISCTE 
criteria statements. Lastly, the authors noted that the competency statements must be “...treated 
as tentative predictors to professional effectiveness, and are subjected to continual validation 
procedures” (p.12). This criterion helped the researchers understand that while current policy and 
practice may limit a certain OISCTE criterion from achieving maximum effectiveness, it should 
not limit the criterions’ inclusion if inclusion equated to overall professional effectiveness as it 
relates to the operation of secondary-level CTE. Additionally, this criterion was the impetus for 
validating the OISCTE using a group of CTE leaders. 

Review of Related Literature 

The regulation, administration, and evaluation of state CTE programs vary widely 
between states. For example, most secondary-level CTE programs are administered within 
offices of their Department of Education, but seven states operate their CTE programs through 
“.. .an agency other than the State Department of Education.. .’’(NASDCTE, 2010, p. 2). 
Additionally, six states politically appoint their state director of CTE rather than engage in the 
hiring of career employees (2010). While the researchers do believe these variances in 
administration have an effect on the operation of secondary-level CTE, the criteria they sought to 
define transcended these ever-changing administrative structures. In addition, much of the 
literature concerning this topic focused on one level within the administrative structure. For 
example, research has been done related to the objectives and tasks of state administrators (Ruff, 
1981; Stevenson, 1982) as well as local administrators (Magisos & Schroeder, 1974; Tennessee 
State Board of Education, 1975; Clark, Farmer, & Welch, 2010). Additionally, research has been 
done related to the objectives and tasks of teacher educators (Stevenson, 1963; Norton, 1977; 
Norton & Harrington, 1987). Lastly, another portion of the literature is made up of strategic 
plans for the operation of CTE. These time-sensitive strategic plans, more often known as state 
plans, have vision areas that focus on statewide objectives related to the operation of CTE. While 
promising, these vision areas usually focus on ancillary objectives related to the Perkins Act such 
as the integration of academic curriculum or assessing CTE completers’ content knowledge with 
standardized assessments. 

While all of the aforementioned pieces of literature are significant, the researchers could 
find no published research that defined and validated a broad set of objectives, tasks, and/or 
criteria statements that encompassed all levels of CTE within a state. As a result, the researchers 
began the process of defining the OISCTE. 

Defining the OISCTE 
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The criteria that make up the OISCTE were derived from a synthesis of four national and 
four state documents (Virginia) related to the regulation, administration, and evaluation of CTE. 
The result was the development of 13 operational criteria statements that were identified as 
embodying the OISCTE. Described in the following sections are the national and state resources 
used to define the OISCTE. 

National-level Resources 

At the national level, four documents assisted the researchers in conceptualizing the “big 
picture” of operating a statewide CTE program: (a) Carl D. Perkins Career and Technical 
Education Act of 2006, (b) National Center for CTE’s vocational administrators’ competency 
studies (1977 and 1987), and (c) the text, Administration of Vocational Education (Wenrich, 
Wenrich, & Galloway, 1988). 

The current federal legislation governing CTE programs, the 2006 Perkins Act, provided 
documentation of the intended purpose of the Act and outlined the required and permissible uses 
of the Perkins funds. This document helped to define what state-level processes were considered 
of federal importance, thus important to include in the OISCTE. 

Further, at the national level, in response to the need to identify competencies required of 
vocational administrators, the Occupational and Adult Education Branch of the U.S. Office of 
Education, under provisions of part C—Research on the Vocational Education Amendments of 
1968, funded a project entitled “Development of Competency Based Instructional Materials for 
Local Administrators of Vocational Education” during the period of 1975-77. This project had 
two major objectives: (a) to conduct research to identify and nationally verify the competencies 
considered important to local administrators of vocational education and (b) to develop and field 
test a series of prototypic competency-based instructional packages and a user’s guide (Norton, 
1977). 


A study to update the 1977 study was completed in 1987. The purpose of the study was to 
identify and nationally verify the competencies needed by administrators of secondary and 
postsecondary vocational and technical education programs and compared the results to the 1977 
study. The survey instrument used in the 1987 study was a three-day Developing a Curriculum 
(DACUM) workshop. The DACUM committee identified 210 tasks that were clustered into 12 
duty areas. A validation questionnaire was sent to 188 administrators, 128 of whom returned 
usable questionnaires. Of the 210 task statements included on the questionnaire, 201 were 
verified as being important (Norton & Harrington, 1987). The ultimate outcome of these two 
studies was to rigorously identify the important competencies needed by CTE administrators 
and, in the context of this study, serves as a rationale for the criteria identified within the 
OISCTE. 

Lastly, the text, Administration of Vocational Education, by Wenrich, Wenrich, and 
Galloway (1988) served as another source for the development of the 13 criteria statements that 
embody the OISCTE. This text addressed, among other things, “■■.administrative structures for 
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delivering vocational education and key aspects of the major functions that any educational 
administrator must perform” (1988, p. 1). 

State-level Resources 

At the state level, four documents provided the researchers with a local-level perspective. 
The four documents were the: (a) Regulations Governing CTE in Virginia (Virginia Board of 
Education, 2001), (b) 1988 Vocational Education Evaluation of Virginia (VEEVA), (c) 

Arizona’s local program evaluation model, and (d) correspondence from the state director of 
Virginia, Elizabeth Russell. 

The Virginia Regulations Governing CTE developed by the Virginia Board of Education 
in 2001 (Virginia Board of Education, 2001), as well as e-mail communication with the director 
of the Office of Career and Technical Education Services from the Department of Education 
assisted the researchers in identifying the CTE administrative structure within their home state 
(personal communication, December 19, 2008). 

Additionally, a review of the Vocational Education Evaluation of Virginia, which was the 
local program evaluation used prior to the enactment of the 1990 Perkins Act, provided an 
overall perception of the what the state, at that time, thought were important areas to evaluate 
when examining the local secondary CTE agencies. 

Lastly, a draft copy of Arizona’s Career and Technical Education Program Review Guide 
(Assessment and Action Plan) which was prepared by Elliot and Molina (2007) in collaboration 
with the University of Arizona’s Department of Career and Technical Education, the Division of 
CTE at Arizona’s Department of Education, and Arizona’s Agriculture Teacher’s Association 
Curriculum Committee, was examined for a more current perspective of secondary CTE local 
program evaluation. Although specifically designed to “provide information to local personnel 
for redirection of the program to meet the present and future needs of agriculture students” and to 
“serve as a model for reviewing all existing [secondary agriculture] programs and guide a new or 
expanding programs” (Elliott & Molina, 2007, p. 2), the program components assessed in this 
program review guide embody more than secondary agriculture programs. 

The 13 Criteria of the Operational Infrastructure of CTE in Virginia 

Table 1 lists each of the criteria statements that embody the OISCTE as well as the 
identification of previously mentioned national and state resources used in the synthesis of each 
criterion statement. 

Table 1 

The Operational Infrastructure of Secondary-level CTE - Criteria and Related Resources 
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Criteria Statements 



1 Provide state coordination, leadership, and technical x x x x x 

assistance for local systems to identify, interpret, and 

comply with local program standards developed and 
accepted by the CTE profession as well as federal and 
state regulations. 

2 Provide relevant professional and leadership x x x x x x 

development to teachers, administrators, and teacher 

educators. 

3 Develop new programs and curricula to respond to x x x x x x 

current and projected occupational needs. 

4 Formulate new and improve existing operational policies x x x x x x x 

(standards) at the local level. 

5 Develop, improve, and maintain quality CTE teacher x 

education programs at the bachelors, master’s, and 
doctoral level. 

6 Conduct innovative research projects to assist in meeting x x x 

unmet CTE needs 

7 Educate and recruit quality CTE teachers and leaders x x x x x 

8 Operate and maintain local CTE facilities and equipment x x x x x x 

9 Develop and execute external and internal procedures for x x x x x x 

program planning, development and evaluation 

procedures for local CTE programs. 

10 Develop, improve, and maintain state and local x x x x x x 

community relations (e.g., advisory committee, CTSO, 

other workforce agencies). 

11 Develop, improve, and maintain statewide professional x x x x x 

associations and CTE teachers and administrators at all 

levels (local, state, and universities). 
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Resource 7 



Table 1 


The Operational Infrastructure of Secondary-level CTE - Criteria and Related Resources 
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d The 2006 Perkins Act (U.S. Congress, 2006) 

Administrator Competency Studies (Norton & Harrington, 1987; Norton, 1977) 
c Administration of Vocational Education (Wenrich, Wenrich, & Galloway, 1988) 
d Regulations Governing CTE in Virginia (Virginia Board of Education, 2001) 

Vocational Education Evaluation of Virginia (Virginia DOE, 1989) 

£ 

Arizona Local Program Evaluation Model (Elliot & Molina, 2007) 

g Email correspondence with Elizabeth Russell Director of CTE in Virginia (E. Russell, personal 
communication, November 11, 2008) 


Findings 


Out of 81 potential participants contacted, 54 initially agreed to participate in the Policy 
Delphi study. Forty-two of the participants completed the Round 1 survey that housed, among 
other information, the validation of the OISCTE. This resulted in a 78% Round 1 participation 
rate. Of the 42 that validated the OISCTE, 21 were local CTE administrators, 12 were state-level 
administrators, and 9 were university-level teacher educators and researchers. The validation of 
the OISCTE was strengthened by the fact that the Delphi participants (n=42) that validated the 
OISCTE had an average of 30.4 years (3=8.4) of cumulative work experience in CTE within 
Virginia. 


The 42 participants validated each of the 13 criterion statements using a five-point Likert 
scale of agreement. As evident in Table 2, all 13 criteria statements had a mean above 4.0. In 
addition, a post-analysis of the data showed an interquartile range (IQR) of 1 or less. The IQR is 
one of many ways in which consensus concerning a rating can be determined (Rayens & Hahn, 
2000). An IQR of 1 on a five-point scale denotes no more than a 20% difference between the 
25 th and 75 th percentiles. While there are not set values for determining consensus, an “IQR that 
is 20% of the rating scale appears to be a conservative but acceptable criterion for determining 
consensus” (Plinske & Packard, 2010, p. 296). 
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Table 2 


OISCTE Validation Results 



Criteria Statements 

Median 

Mean 

Q 

m 

a 

HH 

1 

Provide state coordination, leadership, and technical 
assistance for local systems to identify, interpret, and comply 
with local program standards developed and accepted by the 
CTE profession as well as federal and state regulations. 

5 

4.79 

.606 

0 

2 

Provide relevant professional and leadership development to 
teachers, administrators, and teacher educators. 

5 

4.79 

.520 

0 

3 

Develop new programs and curricula to respond to current 
and projected occupational needs. 

5 

4.81 

.397 

0 

4 

Formulate new and improve existing operational policies 
(standards) at the local level. 

5 

4.36 

.759 

1 

5 

Develop, improve, and maintain quality CTE teacher 
education programs at the bachelors, master’s, and doctoral 
level. 

5 

4.55 

.889 

1 

6 

Conduct innovative research projects to assist in meeting 
unmet CTE needs 

5 

4.48 

.804 

1 

7 

Educate and recruit quality CTE teachers and leaders 

5 

4.48 

.397 

0 

8 

Operate and maintain local CTE facilities and equipment 

5 

4.62 

.661 

1 

9 

Develop and execute external and internal procedures for 
program planning, development and evaluation procedures for 
local CTE programs. 

5 

4.55 

.670 

1 

10 

Develop, improve, and maintain state and local community 
relations (e.g., advisory committee, CTSO, other workforce 
agencies). 

5 

4.64 

.618 

1 

11 

Develop, improve, and maintain statewide professional 
associations and CTE teachers and administrators at all levels 
(local, state, and universities). 

5 

4.50 

.634 

1 
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Table 2 


OISCTE Validation Results 


c 

*3 

Criteria Statements ^ 

Mean 

Q 

a 

HH 

12 Develop, grow and maintain student services (e.g.. Student 5 

recruitment and admissions, placement, and guidance 
services) 

4.55 

.739 

1 

13 Develop, improve, and maintain fiscal management of local 5 

CTE programs. 

4.33 

1.004 

1 


Conclusion and Recommendations 

The findings of this study provide CTE leaders, administrators, and researchers with a set 
of operational criteria that go beyond the ancillary objectives that are the rationale for many 
changes in federal and state-level CTE policy. While the sample size was not large (n=42) and 
only represented CTE administrators and teacher educators in one state (Virginia), the 
researchers believe that the strong consensus as to the inclusion of the criteria statements, as well 
as the average participant tenure within the field of secondary-level CTE leadership (30.4 years), 
increases the generalizability of the OISCTE criteria statements. However, validating the 
OISCTE on a national level could confirm the generalizability of the OISCTE. 

Potential Applications of the OISCTE 

There are at least three potential applications of the OISCTE. First, this framework could 
be used to assess the impact changes within a state’s CTE program has on its ability to operate its 
CTE program. For example, the OISCTE was developed by the researchers to measure the 
impact the 1990 Carl D. Perkins Act within-state funding fonnula change had on Virginia’s 
ability to operate CTE. Coupled with the Policy Delphi technique, which is “an organized 
method for correlating views and information pertaining to specific policy area and for allowing 
the participants representing such views and information the opportunity to react to and assess 
differing viewpoints” (Turoff, 2002, p. 83), the criteria that embody the OISCTE were ideal 
variables for measuring the effect of the change in the within-state funding formula. The findings 
from that study show that it negatively impacted all but one of the criteria statements. Most 
notably, the change in the within-state funding formula adversely effected “CTE teacher 
education, state-level CTE research initiatives, and state-level CTE’s ability to provide localities 
with hands-on technical assistance, professional and leadership development, and coordination. 
The findings also suggested the legislative changes negatively altered the manner in which 
program evaluation occurred with the state” (Manley, 2010, p. 186). 
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Another potential application of the OISCTE would be the development of state-level 
assessments that measure a state’s efforts and ability to develop, maintain, and grow their 
OISCTE. These 13 assessments, one for each criteria statement, could be used to assess the 
operational performance of a state’s CTE program. Additionally, OISCTE assessments would 
enable higher-level associations and institutions, such as the federal government, determine with 
more validity, which states have strong CTE programs. States that score particularly high on the 
OISCTE assessments could be examined for potential modeling. 

Lastly, the OISCTE could be used by state-level CTE administration when planning how 
to develop, grow, and maintain ancillary priorities often set forth by new federal and state 
regulations. Keeping the OISCTE criteria in mind while allocating resources to these new 
priorities may ensure that the underlying framework of their CTE program continues to operate 
efficiently and effectively. 

Concluding Remarks 

In closing, examining administrative and policy-related changes through the OISCTE 
lens compels one to ask important questions. For example, how can best practices be researched 
and disseminated without additional support for research at all levels? How can we recruit and 
train quality CTE teachers without adequate support for teacher education institutions? How can 
local-level administrators and teachers effectively integrate academic instruction into CTE on a 
wide scale without increased support for statewide professional development and technical 
assistance? 

The ability to understand the big picture is one of the workplace readiness skills all CTE 
teachers within Virginia, and other states, are asked to emphasize and assess in their instruction. 
The researchers believe it’s important that leaders in CTE not lose sight of the bigger picture. 
State and local leaders must continue to fight for the preservation of funds (federal, state, and 
local) to accomplish the operational criteria identified within the OISCTE. Failure to do so will 
make the completion of other ancillary objectives and tasks put forth by new legislation and 
regulations difficult, if not impossible, to effectively accomplish in a timely manner. 
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